Log Analysis and Visulization¶

import utils
import os
from os import path
import pickle
import pandas as pd

if __name__ == "__main__":
    
    with open('./tmp/shard.pkl', 'rb') as f:
        shard = pickle.load(f)
    with open('./tmp/param.pkl', 'rb') as f:
        param = pickle.load(f)
        
    date = shard[0].iloc[0]['timestamp'].strftime("%Y-%m-%d %H:%M:%S").split()[0].replace("-","_")     
    fig_dir = "../../graphs/test_logs/ostn/" + date + "/part/"
    html_dir = "../../docs/graphs/test_logs/ostn/" + date + "/part/"
    
    data = [html_dir, fig_dir]
    for directory in data:
        if not path.exists(directory):
            try:
                os.makedirs(directory)
            except:
                print("Could not make directory")
                exit(1)
    
    utils.set_config(param)
    config = utils.config
    ## get the index of the new chain started
    index = []
    for s in shard:
        i = s[s['time_per_block'] < 0].index.tolist()   
        if len(i) == 0:
            break
        i.insert(0,0)
        i.append(len(s))
        index.append(i)
    
    if len(index) == 0:
        new = pd.concat(shard)
        new.reset_index(inplace = True, drop = True)
        utils.visualization(new, fig_dir, html_dir, config)
    else:    
        for i in range(len(index[0])-1):
            df = []
            for s in range(len(shard)):
                df.append(shard[s].iloc[index[s][i]+1:index[s][i+1]])
            new = pd.concat(df)

            fig_path = fig_dir + "chain_" + str(i) + "/"
            if not path.exists(fig_path):
                os.makedirs(fig_path)

            utils.visualization(new, fig_path, html_dir, config)

Statistics summary for shard 0
==================================
Total data points: 4998
              size           gas  transaction_per_second  time_per_block
mean   2208.577231  4.918158e+04                0.003746        8.591437
std    5765.645008  4.313001e+05                0.020627        0.508820
min     848.000000  0.000000e+00                0.000000        8.000000
25%    1160.000000  0.000000e+00                0.000000        8.000000
50%    1277.000000  0.000000e+00                0.000000        9.000000
75%    1315.000000  2.338000e+04                0.000000        9.000000
max   37522.000000  5.415196e+06                0.222222       10.000000

Statistics summary for shard 1
==================================
Total data points: 4998
              size           gas  transaction_per_second  time_per_block
mean   1596.077831      8.403361                 0.00005        8.260904
std    5637.472248    420.041989                 0.00250        0.439172
min     689.000000      0.000000                 0.00000        8.000000
25%     689.000000      0.000000                 0.00000        8.000000
50%     690.000000      0.000000                 0.00000        8.000000
75%     690.000000      0.000000                 0.00000        9.000000
max   36649.000000  21000.000000                 0.12500        9.000000

Statistics summary for shard 2
==================================
Total data points: 4975
              size  gas  transaction_per_second  time_per_block
mean   1526.400201  0.0                     0.0        8.135075
std    5426.295824  0.0                     0.0        0.340955
min     688.000000  0.0                     0.0        8.000000
25%     688.000000  0.0                     0.0        8.000000
50%     688.000000  0.0                     0.0        8.000000
75%     688.000000  0.0                     0.0        8.000000
max   36648.000000  0.0                     0.0        9.000000

Statistics summary for shard 3
==================================
Total data points: 4981
              size  gas  transaction_per_second  time_per_block
mean   1556.244128  0.0                     0.0        8.069062
std    5513.918962  0.0                     0.0        0.252793
min     689.000000  0.0                     0.0        8.000000
25%     690.000000  0.0                     0.0        8.000000
50%     690.000000  0.0                     0.0        8.000000
75%     690.000000  0.0                     0.0        8.000000
max   36649.000000  0.0                     0.0        9.000000

Features vs Time
PNG saved in ../../graphs/test_logs/ostn/2020_04_16/part/
==================================

HTML saved in

HTML saved in

HTML saved in

HTML saved in

HTML saved in

HTML saved in